Language Model Pre-training on True Negatives

Discriminative pre-trained language models (PrLMs) learn to predict original texts from intentionally corrupted ones. Taking the former text as positive and latter negative samples, PrLM can be trained effectively for contextualized representation. However, training of such a type PrLMs highly relies on quality automatically constructed samples. Existing simply treat all equal without any examination, which actually lets resulting model inevitably suffer false issue where is carried out pseudo-negative data leads less efficiency robustness in PrLMs. In this work, basis defining discriminative that has been ignored long time, we design enhanced pre-training methods counteract predictions encourage true negatives by correcting harmful gradient updates subject predictions. Experimental results GLUE SQuAD benchmarks show our counter-false-negative indeed bring about better performance together with stronger robustness.